Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[libbeat][reader][parquet] - Updated Apache Arrow library from v11 to v12.0.1 #35640

Merged
merged 16 commits into from
Jun 12, 2023
Merged

[libbeat][reader][parquet] - Updated Apache Arrow library from v11 to v12.0.1 #35640

merged 16 commits into from
Jun 12, 2023

Conversation

ShourieG
Copy link
Contributor

@ShourieG ShourieG commented Jun 1, 2023 •
edited
Loading

Type of change

Please label this PR with one of the following labels, depending on the scope of your change:

  • Bug

What does this PR do?

This upgrades the apache arrow library inside x-pack libbeat parquet reader from v11 -> v12.0.1

Why is it important?

This upgrade was made in hope of eliminating the cross-compilation errors during building of packages which were using
this reader. Currently the CI is failing due to cross compilation issues arising from errors in the library on certain platforms.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [] I have made corresponding changes to the documentation
    - [] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc

Observations:

The coss-compilation errors observed are:

>> Building using: cmd='build/mage-linux-arm64 golangCrossBuild', env=[CC=oa64-clang, CXX=oa64-clang++, GOARCH=arm64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-arm64]
go build github.com/aws/aws-sdk-go-v2/service/ec2: /usr/local/go/pkg/tool/linux_arm64/compile: signal: killed
Error: running "go build -o build/golang-crossbuild/filebeat-darwin-arm64 -buildmode pie -trimpath -ldflags -s -X github.com/elastic/beats/v7/libbeat/version.buildTime=2023-06-01T11:34:49Z -X github.com/elastic/beats/v7/libbeat/version.commit=1b8bff540d3ceeb77421287b98e78f7ca15251ce" failed with exit code 1
Error: failed building for darwin/arm64: exit status 1
failed building for darwin/arm64: exit status 1
>> golangCrossBuild: Building for linux/arm64
>> Building using: cmd='build/mage-linux-arm64 golangCrossBuild', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64]
go build github.com/aws/aws-sdk-go-v2/service/ec2: /usr/local/go/pkg/tool/linux_arm64/compile: signal: killed
Error: running "go build -o build/golang-crossbuild/filebeat-linux-arm64 -buildmode pie -trimpath -tags=withjournald -ldflags -s -X github.com/elastic/beats/v7/libbeat/version.buildTime=2023-06-01T11:42:37Z -X github.com/elastic/beats/v7/libbeat/version.commit=1b8bff540d3ceeb77421287b98e78f7ca15251ce" failed with exit code 1
Error: failed building for linux/arm64: exit status 1
failed building for linux/arm64: exit status 1
# github.com/apache/arrow/go/v11/internal/utils
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:76:4: undefined: TransposeInt8Int8
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:78:4: undefined: TransposeInt8Int16
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:80:4: undefined: TransposeInt8Int32
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:82:4: undefined: TransposeInt8Int64
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:84:4: undefined: TransposeInt8Uint8
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:86:4: undefined: TransposeInt8Uint16
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:88:4: undefined: TransposeInt8Uint32
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:90:4: undefined: TransposeInt8Uint64
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:95:4: undefined: TransposeInt16Int8
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:97:4: undefined: TransposeInt16Int16
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:97:4: too many errors
go build github.com/aws/aws-sdk-go-v2/service/ec2: /usr/local/go/pkg/tool/linux_amd64/compile: signal: killed
# github.com/apache/arrow/go/v11/internal/utils
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:76:4: undefined: TransposeInt8Int8
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:78:4: undefined: TransposeInt8Int16
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:80:4: undefined: TransposeInt8Int32
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:82:4: undefined: TransposeInt8Int64
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:84:4: undefined: TransposeInt8Uint8
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:86:4: undefined: TransposeInt8Uint16
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:88:4: undefined: TransposeInt8Uint32
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:90:4: undefined: TransposeInt8Uint64
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:95:4: undefined: TransposeInt16Int8
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:97:4: undefined: TransposeInt16Int16
/go/pkg/mod/github.com/apache/arrow/go/v11@v11.0.0/internal/utils/transpose_ints_def.go:97:4: too many errors
go build github.com/aws/aws-sdk-go-v2/service/ec2: /usr/local/go/pkg/tool/linux_amd64/compile: signal: killed
go build github.com/aws/aws-sdk-go-v2/service/ec2: /usr/local/go/pkg/tool/linux_amd64/compile: signal: killed

CI Link : https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats/detail/PR-35578/8/pipeline#562

It seems that the issue is reported here apache/arrow#34784
And it's also stated that upgrading to v12 only fixes the transpose errors, but another error - an integer over flow error still remains and is supposed to get fixed in 12.0.1

Overflow issue - apache/arrow#35861

update: library has been updated to 12.0.1

Related issues

@ShourieG ShourieG requested review from a team as code owners June 1, 2023 13:54
@ShourieG ShourieG requested review from rdner and cmacknz June 1, 2023 13:54
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 1, 2023
@ShourieG ShourieG self-assigned this Jun 1, 2023
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 1, 2023
@ShourieG ShourieG added needs_team Indicates that the issue/PR needs a Team:* label ci-reported Issues that have been automatically reported from the CI labels Jun 1, 2023
@mergify
Copy link
Contributor

mergify bot commented Jun 1, 2023

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @ShourieG? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 1, 2023
@botelastic
Copy link

botelastic bot commented Jun 1, 2023

This pull request doesn't have a Team:<team> label.

@ShourieG ShourieG requested a review from andrewkroh June 1, 2023 14:02
@ShourieG ShourieG mentioned this pull request Jun 1, 2023
Merged
5 tasks
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 1, 2023 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-06-12T06:40:23.655+0000

  • Duration: 105 min 10 sec

Test stats 🧪

Test Results
Failed 0
Passed 27337
Skipped 2009
Total 29346

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@andrewkroh
Copy link
Member

Currently the CI is failing due to cross compilation issues arising from errors in the library on certain platforms like linux.

Could you point us to those errors and include them in the description for posterity.

@ShourieG
Copy link
Contributor Author

ShourieG commented Jun 1, 2023

Currently the CI is failing due to cross compilation issues arising from errors in the library on certain platforms like linux.

Could you point us to those errors and include them in the description for posterity.

I have updated the description.

@ShourieG ShourieG added blocked and removed blocked labels Jun 5, 2023
@ShourieG ShourieG removed the ci-reported Issues that have been automatically reported from the CI label Jun 7, 2023
@ShourieG ShourieG changed the title [libbeat][reader][parquet] - Updated Apache Arrow library from v11 -> v12 [libbeat][reader][parquet] - Updated Apache Arrow library from v11 -> v13 Jun 7, 2023
@ShourieG ShourieG changed the title [libbeat][reader][parquet] - Updated Apache Arrow library from v11 -> v13 [libbeat][reader][parquet] - Updated Apache Arrow library from v11 to v13 Jun 7, 2023
@ShourieG ShourieG changed the title [libbeat][reader][parquet] - Updated Apache Arrow library from v11 to v13 [libbeat][reader][parquet] - Updated Apache Arrow library from v11 to v13 dev Jun 7, 2023
@ShourieG
Copy link
Contributor Author

ShourieG commented Jun 7, 2023 •
edited
Loading

@andrewkroh updated the library to v12.0.1 with the json test file changes.

@ShourieG ShourieG changed the title [libbeat][reader][parquet] - Updated Apache Arrow library from v11 to v13 dev [libbeat][reader][parquet] - Updated Apache Arrow library from v11 to v12.0.1 Jun 7, 2023
@ShourieG
Copy link
Contributor Author

ShourieG commented Jun 9, 2023 •
edited
Loading

@andrewkroh I've shortened the vpc flow json by reducing it to 4 records, formatted the json files and adjusted the tests accordingly. We cannot alter the main vpc parquet file as that is used in benchmarks also.

@ShourieG ShourieG merged commit 54b5d97 into elastic:main Jun 12, 2023
@ShourieG ShourieG deleted the libbeat/parquet branch June 12, 2023 10:00
faec added a commit to faec/beats that referenced this pull request Jun 14, 2023
@faec
Revert "[libbeat][reader][parquet] - Updated Apache Arrow library fro…
8bffc3d 
…m v11 to v12.0.1 (elastic#35640)"

This reverts commit 54b5d97.
@faec faec mentioned this pull request Jun 14, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@efd6 efd6 efd6 left review comments

@andrewkroh andrewkroh andrewkroh approved these changes

@belimawr belimawr belimawr approved these changes

@rdner rdner Awaiting requested review from rdner rdner was automatically assigned from elastic/elastic-agent-data-plane

@cmacknz cmacknz Awaiting requested review from cmacknz cmacknz was automatically assigned from elastic/elastic-agent-data-plane

Assignees

@ShourieG ShourieG

Labels
8.9 candidate bugfix
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@ShourieG @elasticmachine @andrewkroh @belimawr @efd6